Weighted ensemble boosting method for classifier combination and feature selection

ABSTRACT

A method constructs a strong classifier from weak classifiers by combining the weak classifiers to form a set of combinations of the weak classifiers. Each combination of weak classifiers is boosted to determine a weighted score for each combination of weak classifiers, and combinations of weak classifiers having a weighted score greater than a predetermined threshold are selected to form the strong classifier.

FIELD OF THE INVENTION

This invention relates generally to computer implemented classifiers,and more specifically to strong classifiers that are constructed bycombining multiple weak classifiers.

BACKGROUND OF THE INVENTION

Recognition of activities and objects plays a central role insurveillance and computer vision applications, see A. F. Bobick,“Movement, activity, and action: The role of knowledge in the perceptionof motion,” Royal Society Workshop on Knowledge-based Vision in Man andMachine, 1997; Aggarwal et al., “Human motion analysis: A review,”Computer Vision and Image Understanding, vol. 73, no. 3, pp. 428-440,1999; and Nevatia et al., “Video-based event recognition: activityrepresentation and probabilistic recognition methods,” Computer Visionand Image Understanding, vol. 96, no. 2, pp. 129-162, November 2004.

Recognition, in part, is a classification task. The main difficulty inevent and object recognition is the large number of events and objectclasses. Therefore, systems should be able to make a decision based oncomplex classifications derived from a large number of simplerclassifications tasks.

Consequently, many classifiers combine a number of weak classifiers toconstruct a strong classifier. The main purpose of combining classifiersis to pool the individual outputs of the weak classifiers as componentsof the strong classifier, the combined classifier being more accuratethan each individual component classifier.

Prior art methods for combining classifiers include methods that applysum, voting and product combination rules, see Ross et al., “Informationfusion in biometrics,” Pattern Recognition Letters, vol. 24, no. 13, pp.2115-2125, 2003; Pekalska et al., “A discussion on the classifierprojection space for classifier combining,” 3rd International Workshopon Multiple Classifier Systems, Springer Verlag, pp. 137-148, 2002;Kittler et al., “Combining evidence in multimodal personal identityrecognition systems,” Intl. Conference on Audio- and Video-BasedBiometric Authentication, 1997; Tax et al., “Combining multipleclassifiers by averaging or by multiplying?” Pattern Recognition, vol.33, pp. 1475-1485, 2000; Bilmes et al., “Directed graphical models ofclassifier combination: Application to phone recognition,” Intl.Conference on Spoken Language Processing, 2000; and Ivanov, “Multi-modalhuman identification system,” Workshop on Applications of ComputerVision, 2004.

SUMMARY OF THE INVENTION

One embodiment of the invention provides a method for combining weakclassifiers into a strong classifier using a weighted ensemble boosting.The weighted ensemble boosting method combines Bayesian averagingstrategy with a boosting framework, finding useful conjunctive featurecombinations of the classifiers and achieving a lower error rate thanthe prior art boosting process. The method demonstrates a comparablelevel of stability with respect to the composition of a classifierselection pool.

More particularly, a method constructs a strong classifier from weakclassifiers by combining the weak classifiers to form a set ofcombinations of the weak classifiers. Each combination of weakclassifiers is boosted to determine a weighted score for eachcombination of weak classifiers, and combinations of weak classifiershaving a weighted score greater than a predetermined threshold areselected to form the strong classifier.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow diagram of a method for constructing a strongclassifier from a combination of weak classifiers according to anembodiment of the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

FIG. 1 shows a method for constructing a strong classifier 109 from weakclassifiers (A, B, C) 101 according to an embodiment of the invention.The weak classifiers are combined 110 to produce a set of combinedclassifiers 102. Then, a boosting process 120 is applied to the set ofcombined classifiers to construct the strong classifier 109.

Weak Classifiers

The weak classifiers can include binary and multi-class classifiers. Abinary classifier determines whether a single class is recognized ornot. A multi-class classifier can recognize several classes.

An output of each weak classifier can be represented by posteriorprobabilities. Each probability indicates how certain a classifier isabout a particular classification, e.g., the object identity. Inaddition, each weak classifier can be associated with a confidencematrix. The confidence matrix indicates how well the classifier performsfor a particular class. The confidence matrices are obtained by trainingand validating the classifiers with known or ‘labeled’ data.

Combining

The combining step can include all possible linear combinations 102′ ofthe weak classifiers, as well as various non-linear combinations 102″For example, six weak classifiers can yield over 500 combinations.

The combining 110 can also use an adaptation of an approximate Bayesiancombination. The Baysian combination uses some measure of classifierconfidence to weigh the prediction probabilities of each weak classifierwith respect to an expected accuracy of the weak classifier for each ofthe classes, see Ivanov, “Multi-modal human identification system,”Workshop on Applications of Computer Vision, 2004; and Ivanov et al.,“Using component features for face recognition,” InternationalConference on Automatic Face and Gesture Recognition, 2004, bothincorporated herein by reference.

More particularly, an output of weak classifier, λ, is viewed as arandom variable, {tilde over (ω)} taking integer values from 1 to K,i.e., the number of classes. If, for each classifier, the probabilitydistribution over values of a true class label ω is available for agiven classifier prediction, P_(λ)(ω|{tilde over (ω)}), then theapproximate Bayesian combination can be derived via marginalization ofindividual class predictions of each weak classifier:${P_{a}\left( \omega_{i} \middle| x \right)} = {\sum\limits_{k = 1}^{K}{w_{k}\underset{\underset{P_{k}{({\omega_{i}|x})}}{︸}}{{\sum\limits_{j = 1}^{J}{P_{k}\left( \omega_{i} \middle| {\overset{\sim}{\omega}}_{j} \right){P_{k}\left( {\overset{\sim}{\omega}}_{j} \middle| x \right)}}},}}}$where P_(k)({tilde over (ω)}|x) is the prediction probability of theweak classifier, and w_(k) is a weight of each weak classifier. Equationweights a prediction of each classifier in accordance to the confidencematrix associated with the class.

The combinations in the set 102 are formed for singles, pairs, triples,etc., of the weak classifiers 101. The non-linear transformation isaccording to:${{P_{n}^{\beta}\left( \omega_{i} \middle| x \right)} = \frac{\exp\left( {\beta{\sum\limits_{j \in S_{n}}{P_{j}\left( \omega_{i} \middle| x \right)}}} \right)}{\sum\limits_{c = 1}^{C}{\exp\left\lbrack \left( {\beta{\sum\limits_{k \in S_{n}}{P_{k}\left( \omega_{c} \middle| x \right)}}} \right) \right\rbrack}}},$where P_(k)(ω_(j)|x) is a weighted weak classifier according to anon-linear weight β and S_(n) is an n^(th) classifier combination. Foran exhaustive enumeration of combinations, the total number of thetuples for every value of β is given by the following relation:$N = {{\sum\limits_{n = 1}^{K}\begin{pmatrix}K \\n\end{pmatrix}} = {2^{K} - 1.}}$where K is the number of weak classifiers and N is the number of tuples.That is, if 8 different values of β are used to form combinations of 6classifiers, the total number of these combinations 102 comes to 504.

Boosting

As stated above, the strong classifier 109 is derived from the set ofcombined weak classifiers 102. The boosting essentially ‘discards’combinations in the set that have low ‘weights’, e.g., weights less thansome predetermined threshold or zero, and keeps the combinations thatare greater than the predetermined threshold. The number of elements inthe strong classifier can be controlled by the threshold.

The method adapts the well known AdaBoost process, Freund et al., “Adecision-theoretic generalization of on-line learning and an applicationto boosting,” Computational Learning Theory, Eurocolt '95, pp. 23-37,1995, incorporated herein by reference.

The AdaBoost process trains each classifier in a combination withincreasingly more difficult data, and then uses a weighted score. Duringthe training, the combined classifiers are examined, in turn, withreplacement. At every iteration, a greedy selection is made. Thecombined classifier that yields a minimal error rate on datamisclassified during a previous iteration is selected, and the weight isdetermined as a function of the error rate. The AdaBoost processiterates until one of the following conditions is met: a predeterminednumber of iterations has been made, a pre-determined number ofclassifiers have been selected, the error rate decreases to apre-determined threshold, or no further improvement to the error ratecan be made.

Formally, a probability distribution over the classes can be expressedas a weighted sum of the scores:${{P_{b}\left( \omega_{i} \middle| x \right)} \propto {\sum\limits_{k = 1}^{K}{W_{k}\left\lbrack {{{argmax}\quad{P_{k}\left( \overset{\sim}{\omega} \middle| x \right)}} = i} \right\rbrack}}},$where the weight W_(k) is the aggregate weight of the k^(th) classifier:$W_{k} = {\sum\limits_{t = 1}^{T}{w_{t}\left\lbrack {f_{t} = f_{k}} \right\rbrack}}$This equation states that the weight of the k^(th) classifier is the sumof weights of all instances t of the classifier, where the classifierf_(k) is selected by the process.

Feature stacking can then use the strong classifier trained on theoutputs of weak classifiers stacked into a single vector. That is, theinput for the strong classifier, {tilde over (x)}, is formed as follows:{tilde over (X)}=(P ₁(ω|x)^(T) , P ₂(ω|x)^(T) , . . . , P_(K)(ω|x)^(T))^(T),and then the strong classifier is trained on pairs of data, (X_(i),Y_(i)), where Y_(i) is the class label of the i^(th) data point.

Although the invention has been described by way of examples ofpreferred embodiments, it is to be understood that various otheradaptations and modifications may be made within the spirit and scope ofthe invention. Therefore, it is the object of the appended claims tocover all such variations and modifications as come within the truespirit and scope of the invention.

1. A computer implemented method for constructing a strong classifier,comprising: selecting a plurality of weak classifiers; combining theweak classifiers to form a set of combinations of the weak classifiers;boosting each combination of the weak classifiers to determine aweighted score for each combination of the weak classifiers; andselecting combinations of the weak classifiers having a weighted scoregreater than a predetermined threshold to form the strong classifier. 2.The method of claim 1, in which the weak classifiers include binary andmulti-class classifiers.
 3. The method of claim 1, further comprising:representing an output of each weak classifier by a posteriorprobability; and associating each weak classifier with a confidencematrix.
 4. The method of claim 3, in which the combinations includelinear and non-linear combinations of the weak classifiers.
 5. Themethod of claim 3, in which the combining is an approximate Bayesiancombination.
 6. The method of claim 5, in which an output λ of each weakclassifier is a random variable {tilde over (w)} taking integer valuesfrom 1 to K, the number of classes, and a probability distribution overvalues of a true class label ω is P_(λ)(ω|{tilde over (ω)}), and theapproximate Bayesian combination is${{{P_{a}\left( \omega_{i} \middle| x \right)} = {\sum\limits_{k = 1}^{K}{w_{k}\underset{\underset{P_{k}{({\omega_{i}|x})}}{︸}}{\sum\limits_{j = 1}^{J}{{P_{k}\left( \omega_{i} \middle| {\overset{\sim}{\omega}}_{j} \right)}{P_{k}\left( {\overset{\sim}{\omega}}_{j} \middle| x \right)}}}}}},}\quad$where P_(k)({tilde over (ω)}|x) is a prediction probability of the weakclassifier, and w_(k) is a weight of the classifier.
 7. The method ofclaim 6, in which the weight is according to the confidence matrix ofthe class.
 8. The method of claim 6, in which the set of combinations isformed according to${{P_{n}^{\beta}\left( \omega_{i} \middle| x \right)} = \frac{\exp\left( {\beta{\sum\limits_{j \in S_{n}}{P_{j}\left( \omega_{i} \middle| x \right)}}} \right)}{\sum\limits_{c = 1}^{C}{\exp\left\lbrack \left( {\beta{\sum\limits_{k \in S_{n}}{P_{k}\left( \omega_{c} \middle| x \right)}}} \right) \right\rbrack}}},$where P_(k)(ω_(j)|x) is a weighted weak classifier according to anon-linear weight β, and S_(n) is an n^(th) classifier combination.